Section: New Results

Estimator selection and statistical tests

Participant : Sylvain Arlot.

Sylvain Arlot wrote a book chapter about cross-validation in 2018. This text defines all classical cross-validation procedures, and studies their properties for two different goals: estimating the risk of a given estimator, and selecting the best estimator among a given family. For the risk estimation problem, it computes the bias (which can also be corrected) and the variance of cross-validation methods. For estimator selection, it first provides a first-order analysis (based on expectations). Then, it explains how to take into account second-order terms (from variance computations, and by taking into account the usefulness of over-penalization). This allows, in the end, to provide some guidelines for choosing the best cross-validation method for a given learning problem.